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Executive Summary 


California's new Statewide System of Support is grounded in the fundamental 
principles of the Local Control Funding Formula, especially its emphasis on the central 
role of local educators in determining the best approaches to improvement. This report 
examines the early implementation of the System of Support, with a focus on the work 
of the county offices of education (COEs) and the experience of the districts identified for 
differentiated assistance. 


COE and district officials held positive views of the Key shifts in the System of 
Support, particularly the emphasis on support over compliance and the empowerment 
of local districts as decision makers in their own improvement efforts. However, COEs 
differed in the definitions of their role in this new system, depending on the contexts in 
which they worked. These definitions ranged from being the main source of assistance on 
all challenges facing their districts, to being a thought partner primarily for implementing 
improvement processes, to brokering sources of support beyond the COE where COE 
capacity was insufficient to meet district needs. COEs in our case studies had made 
significant efforts to boost their internal capacity to support their districts and had worked 
to break down internal silos between various departments. Nonetheless, while officials 
in Our case study districts appreciated the efforts of their COEs, many experienced the 
support as less than adequate to address their challenges. Both COE and district officials 
raised concerns about the under-resourcing of the support system. 


A major influence on the effectiveness of the System of Support is the Dashboard. 
Districts are identified for differentiated assistance based on their performance on 
the Dashboard, a measurement tool that employs multiple indicators and disaggregates 
the performance of student subgroups. While interviewees found the Dashboard an 
improvement over the previous single-measure Academic Performance Indicator (API), 
they expressed numerous concerns. Among these were that the Dashboard undermines 
the focus needed for sustainable improvement and generally fails to provide data that 
are sufficiently timely, valid, and comprehensible to guide improvement. In addition, 
respondents expressed concerns about the equitable distribution of support and the 
truncated timeline for assistance. 


The System of Support is a notable effort by the state to reinvent accountability 
aligned with the principles of the Local Control Funding Formula. However, it is not yet 
fully a system, as it is under-funded, fails to draw upon the full range of expertise in the 
State, and is not well-aligned with other accountability components of the Local Control 
Funding Formula. The report concludes with five recommendations designed to make 
the System of Support a more robust and comprehensive system to realize the aspirations 
of the Local Control Funding Formula. 
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Introduction: 
A New Role for County Offices of Education 


The Local Control Funding Formula (LCFF), passed by the California Legislature in 
2013, brought more than a new way to fund K-12 education in the state. It also altered 
the Institutional landscape in education by creating a new state agency — the California 
Collaborative for Educational Excellence — and engendering changes in the roles 
of existing agencies, particularly county offices of education (COEs). These changes, 
along with the funding formula itself, were intended to support both equity and system 
improvement to better serve all of California's young people. To the COEs went the 
new responsibility of annually approving their districts’ Local Control Accountability 
Plans (LCAPs), in which districts outline their intended activities and resource allocation 
strategies to meet the eight state priorities delineated in the LCFF legislation.t Also implied 
in this role was a more supportive function for COEs: to help districts create LCAPs that 
would meet the requirements and achieve the goals of the LCFF.* Early reports suggested 
that COEs took up this supportive role to varying degrees. 


Then, in 2017, the counties’ support function was ratcheted up significantly with 
the implementation of a new accountability system and Statewide System of Support. 
Counties were now to be the first line of support for districts identified as needing 
“differentiated assistance” based on the state’s multiple-indicator California School 
Dashboard. This report examines how COEs are interpreting and acting on their role in the 
new System of Support, how those actions are influenced or constrained by state policy 
design and implementation as well as by local context, and how districts are experiencing 
and responding to differentiated assistance from their county offices. Our intent is to 
report emerging patterns and challenges from the system’s first two years in order to 
suggest possible refinements in design and implementation. 


‘The priority areas outlined in the LCFF legislation are 1) basic services (such as appropriate assignment of teachers), 
2) implementation of the state content and performance standards, 3) parent involvement, 4) student achievement, 
5) student engagement, 6) school climate, 7) access to a broad course of study, and 8) other student outcomes. 
* Throughout this report, our focus is on districts as the primary target of support. Technically, Local Education Agencies 
(LEAs) include school districts, charter schools, and county offices of education, and any of these may be the recipients of 
differentiated assistance (note that charters are not in the System of Support until this school year). Our interest, however, 
is in the COE-to-district support relationship. 
> Humphrey, D.C., and Koppich, J.K. (November 2014). Toward a Grand Vision: Early Implementation of California's Local 
Control Funding Formula. Stanford, CA: Policy Analysis for California Education. Access at httos://edpolicyinca.org/ 
publications/toward-grand-vision-early-implementation-californias-local-control-funding-formula. 
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Study Methodology 


This is the sixth in a series of reports produced by the LCFF Research Collaborative, a collection of 
researchers from six organizations and institutions who have been studying LCFF implementation 
since 2013. This multi-method study draws on data from multiple sources: 


e Reviews of state and COE documents and district LCAPs and Dashboards, 
e Interviews with 20 state-level actors (policy makers, agency leaders, and advocates) (Fall 2018) 
e« Asurvey of 46 COE superintendents in a meeting of the California County Superintendents 
Education Services Association (CCSESA) in January 2019 (and previously in October 2017) 
and a separate survey of 72 other leading COE administrators in CCSESA’s Curriculum and 
nstruction Steering Committee (CISC) in May 2019. 
¢ Case studies of five COEs and 14 districts they serve, for a total of: 

¢ 50 interviews with COE Officials and 

e 111 interviews with district (and school) leaders. 


COE case studies were geographically distributed and represented both urban and rural settings 
(one in the rural north, one in the Central Valley, one in the Bay Area, two in southern California 
(both coastal and inland)). Sampled districts had all been identified for differentiated assistance 
in either or both 2017 and 2018. Site visits occurred in spring 2019 (April through June). Analysis 
took place through summer 2019 and included a two-day cross-site analysis research team 
meeting and multiple rounds of feedback during the drafting of the report. 


What is the Statewide System of Support? 


To understand the evolving role of the COE requires an understanding of the 
intent and structures of the new Statewide System of Support, which is itself intended 
to function as part of a much larger, more comprehensive and coherent system to 
improve outcomes for all students, while reducing disparities between targeted groups.* 
In the words of a state policy actor who has been instrumental in the design of the 
new approach: 


4 The LCFF targets additional funding to districts and other local education agencies to increase or improve services to four 
traditionally underserved groups: low-income students, English learners, foster youth, and homeless youth. In addition, 

the accountability Dashboard disaggregates school and district data not only for these students but also for ethnic/racial 
groups and students receiving special education services. 
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This is a system designed to support the ongoing implementation of the 
Local Control Funding Formula and the different pieces that go with it, 
[including] the work of the Local Accountability Plan development, approval, 
and review and the Dashboard... so it’s not the System of Support on its 
own, It’s all part of the accountability structure. 


Within that structure, the stated goal of the System of Support is “to help local 
educational agencies (LEAs) and their schools meet the needs of each student they serve, 
with a focus on building local capacity to sustain improvement and to effectively address 
disparities in opportunities and outcomes.”® 


Levels of Support 


To accomplish that goal, system designers delineated three intended levels of 
support, consistent with the structure laid out in the LCFF legislation: 


¢ Support for all LEAs and schools (Level 1): an array of resources and voluntary 
assistance that all LEAs may use to improve student performance at the LEA and 
school level and narrow disparities among student groups across LCFF priorities. 

¢ Differentiated assistance (Level 2): individually designed assistance for LEAs to 
address identified performance issues for all students and for identified student 
groups. 

¢ Intensive intervention (Level 3): for LEAs with persistent performance issues 
over a specified time period.® 


COEs are involved in both Level 1 and Level 2 supports, but much of the attention 
has been focused on their unique role as the first line of support for districts identified 
for differentiated assistance (DA). COEs must offer that assistance to a school district if 
any student group within that district does not meet performance standards for two or 
more LCFF priority areas, as reported through the district's Dashboard.’ According to the 
July 2017 memo to the State Board of Education: “The consistent approach to assistance 
should focus on supporting LEAs and local stakeholders to identify the underlying cause 
of the challenges and identifying options for addressing that underlying issue.”® While 
the district is not required to accept the assistance offered by the county, the COE must 


° https://www.cde.ca.gov/sp/sw/t1/csss.asp 

5 Level 3 supports have not yet been implemented. 

” California Education Code (EC) Section 52071(c). 

8 California State Board of Education, "Developing an Integrated Statewide System of Support for Local Educational 
Agencies and Schools: Proposed Goals and Characteristics of an Integrated System and Regional Structure for Providing 
Coordinated Support Across State and Federal Programs.” Item #2, July, 2017, https://www.cde.ca.gov/be/ag/ag/ 


documents/agenda201707.pdf 
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make It available. Significantly, “each of the three levels of support represents a type of 
assistance available or provided to LEAs, not a status or label applied to LEAs.”? This is one 
of several key shifts in the approach to accountability and support in the state. 


Intended Shifts in the New State Accountability and Support System 


The new approach to accountability in California has been described by state 
leaders as “radically different” from what came before.*° Based on our review of state 
documents and our discussions with state policymakers, we have identified five intended 
shifts in the new accountability model that distinguish it from prior approaches: 


¢ Froma focus on schools to an emphasis on LEAs (districts) as the unit 
of change. Integral to California's new accountability and support system 
is an emphasis on districts as not only targets for but also the key actors 
in improvement efforts. This district-centric focus is a departure from prior 
accountability systems that centered on identification of and support for 
individual schools. Consistent with Every Student Succeeds Act (ESSA) 
regulations, California continues to identify persistently low-performing 
schools for improvement, but the primary means through which the state 
supports their progress is by mobilizing and supporting their LEA. According 
to a June 2017 State Board of Education memo, “LEAs play the essential role 
in Supporting schools to sustain improvement." 


¢ From punitive accountability to an emphasis on support and continuous 
improvement. Prior accountability structures under the No Child Left Behind 
Act (NCLB) and the California Public School Accountability Act (PSAA) have 
been criticized as trying to blame and shame schools and districts into 
improvement." But commandments and threats have proven ineffective in 
fostering ownership and positive change, especially when individuals and 
systems lack the capacity to identify, understand, and address their underlying 
problems. The new System of Support is based on a different model that 
privileges capacity building and partnerships over command and control. 


¢ From compliance to local decision-making based on local needs. Consistent 
with the LCFF is the new system's incorporation of local control in the design 


° See description of California's Statewide System of Support on the CDE website at https://www.cde.ca.gov/sp/sw/t1/ 
CSSS.aSp 

10 Interviews with multiple state policy makers. 

it Past state and federal laws based accountability on numeric indexes of student achievement, the Academic Performance 
Index (API) and Annual Yearly Progress (AYP) respectively. Consequences — such as removal of staff or governance changes 
— ensued for schools or districts that did not meet the required academic targets set by the state or federal governments. 
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and implementation of assistance. Not only are districts to be the target of help, 
but they are also the intended decision-makers about the nature of the support 
they receive. As one state policymaker explained: 


The whole premise of the design of our system now Is that people have 
to own their problems and take responsibility for them at the local level... 
We're trying to shift it to being more of a partnership [between COEs 
and districts] to figure out the root causes and address them in the most 
effective way. 


¢ From a single criterion or index to a multiple-indicator Dashboard covering 
a broad range of outcomes aligned to the state's eight priority areas. [he 
new system has moved from a blunt identification and labeling approach to a 
more fine-grained monitoring of multiple indicators to allow for identification 
and targeting of specific areas and specific student populations. The designers 
believed that in this way, and with the inclusion of local as well as state 
indicators, assistance can be more responsive to the needs of local districts 
and communities. 


¢ From redundant and contradictory state and federal accountability programs 
to one coordinated and coherent system. California's new approach to 
accountability and support, as described in the state’s approved ESSA plan, 
represents an attempt to integrate federal and state accountability policies. 
In addition, coordination among the various agencies within the state 
has also been stepped up as responsibilities have been clarified and explicit 
avenues for communication and coordination have been established. 


New Roles and Agencies Within the System of Support 


To operationalize these shifts and expand the types and quality of supports 
available, the state has created new roles and actors to support capacity-building and 
improvement at all levels. Particularly important among the new actors are the nine 
Geographic Lead county agencies, whose responsibility it is to build the capacity of 
other COEs In their geographic area, coordinate assistance across the area, and provide 
differentiated assistance to a school district if that district's COE is unable to do so. 


” The current ESSA plan is the product of extended negotiations between state policy makers and the US Department of 
Education to address several fundamental differences between the state and federal laws (e.g. district vs. school as the unit 
of accountability, single index vs. multiple indicators for identification of need). 

4 There are seven geographic areas across the 58 counties. Lead responsibility for each of two of these areas is shared by 


two Geo Lead agencies. 
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In addition, to address the deeply troubling Dashboard results for students with disabilities, 
the state has created seven new leads among the Special Education Local Plan Areas 
(SELPAs).“ There are also expert leads in topical areas (like mathematics or English 
learners) as well as a lead partnership responsible for identifying and spreading effective 
models of community engagement. Coordination among these leads is shared by the 
California Collaborative on Educational Excellence (CCEE) and the California Department 
of Education (CDE).% Though not a state agency, the California County Superintendents 
Educational Services Association (CCSESA) has also played an important role in guiding 
and supporting COEs in their provision of differentiated assistance. 


How are County Offices of Education and the Districts Enacting 
and Responding to the Statewide System of Support? 


A major goal of this study was to understand what actions county offices of 
education are taking to implement differentiated assistance, as well as how the 
affected districts were experiencing the support. Five central findings emerged from our 
investigation. 


1. Both COEs and Local Districts Appear to be Embracing the New Accountability 
and Support Narrative 


On the surveys administered to COE leaders as part of this study, 94 percent of 
county superintendents and an equal percentage of assistant superintendents’ agreed or 
strongly agreed that “The state’s new System of Support is a step in the right direction." 


Our case study COE respondents echoed this perspective. Across our five 
counties, their descriptions of the support system and of their work in it mirrored the 
shifts in approach outlined above, particularly the emphasis on local decision-making 
and responsiveness to local conditions as well as the move towards a more supportive 
rather than punitive approach to accountability. Typical of the responses was that of one 
county improvement team member: “The shift, to me, has been to instead of telling 
people what to do, how can you facilitate the autonomy for them to do better?” Another 
contrasted the current approach with the former accountability system: 


4 See box “Students with Disabilities (SWDs) and the System of Support" for additional details. 

4 |t is important to note that to date the System of Support is comprised only of public agencies, though districts are free 
to seek support from nonprofits and others that they deem most relevant to their needs. 

16 For ease of reference, we refer to the second group of survey respondents as assistant superintendents. Most were in 
this role in their COE, but there were also a few department directors in the group. 
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Local control is really powerful... they [districts] can do things that they've 
known they needed to do for years, but would never have had the 
permission in the old accountability system. But with the ability to make 
local decisions on things that have been identified publicly and with them 
they have the backing of the community. | think it's very powerful. 


This shift from telling to enabling has been reflected in COEs’ efforts to help 
districts establish a continuous improvement process, informed by a CCSESA-developed 
guidance document and tools for DA.” Thus, shortly after the release of the Dashboard 
data in late fall 2018 (or 2017), all case study COEs had sent a letter to each district 
identified for DA offering COE support. This initial overture was followed by meetings 
with the districts (individually and sometimes collectively) to analyze each district's data 
to determine the root causes of one or more of the performance issues revealed on the 
district's Dashboard. The process generally concluded with the COE providing a summary 
of the data and conclusions along with information about available resources. 


Often when the COE staff arrived at the initial meeting with the district, they had 
already conducted a preliminary exploration of the district's data. The root cause analysis 
was then a collaborative process to “dive deeper” into the information to uncover the 
underlying issues. For example, in one district the analysis of the data for students with 
disabilities (SWD) and the questions posed by the COE staff revealed an underlying lack of 
systems such as Student Support Teams, tiered interventions, or systematic data reviews to 
identify and meet the needs of these students. Through this process, explained the COE 
administrator working with this district, “they started looking into their systems, which then 
started driving them back to first best instruction, which then drove them into other arenas 
of the work. What you saw was a slow accumulation of systems being built to better 
support their students.” 


To enable this work, case study COE staff across the board emphasized the 
importance of building positive relationships with their districts as the foundation for 
providing meaningful support. They stressed the need to establish trust by assigning 
individual staff members to each district on an Ongoing basis and by learning as much as 
possible about what the districts were doing, including what they were doing well. Said 
one COE administrator, “It's about relationships... so part of the whole System of Support 
is to learn as much as we can [about our districts].” 


Y Definitions of and approaches to continuous improvement vary, but at its most basic, continuous improvement focuses 
on a well-defined aim and on reducing variation in performance towards that aim. It involves investigation of the systemic 
causes of the identified problem and a disciplined cyclical process of developing, testing, and refining potential solutions 
that can then be implemented broadly to improve performance across the organization or system. The CCSESA tools 
and guidance draw largely on processes developed by the Carnegie Foundation for the Advancement of Teaching and by 


county superintendents. 
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For their part, district respondents in each of the five case study counties 


overwhelmingly acknowledged and appreciated what they perceived as a positive shift in 
their COEs’ approach to accountability and assistance, consistent with the intent of the 
System of Support policy. Typifying these sentiments were the following comments from 


district administrators: 


We're fortunate that the county doesnt come in and try and tell us what 
to do... There arent a whole lot of silver bullets out there. We're trying to 
address our specific issues, but | feel the county works with us... Districts in 
[this county] absolutely get the support they need without them trying to 
come in and tell us how to do it. 


Right now, | see it as a kinder and gentler approach to school improvement... 
Now you are not on an island by yourself. You have a support team. And the 
root cause analysis helps to get to the problem. 


2. District Respondents Were Not Universally Enthusiastic About the Usefulness of 


their County's Support 


In one of our sampled counties, COE administrators stated that they had engaged 


in a root cause analysis with all of their eligible LEAs, but our three sampled districts 


reported difficulty in recognizing it as such. Said one local administrator, “[The COE] did 
have a conversation with [us], but | wouldn't describe it as a root cause analysis.” In two 
other counties, some district respondents felt that the process simply hadn't gone far 
enough or hadn't pushed their thinking beyond what they were already able to do on their 


own. In one case, the district team had analyzed their own data and come up with a 


plan, 


which, in their final meeting with the COE, county staff had simply handed back to them 


along with their original raw data, a nod of approval, and a request for permission to 
the plan as a model for other districts. The local administrators did not find this conc 


Use 
usion 


to the process particularly helpful. Said one, “That is not what we needed. We need your 


[the county's] level of expertise!” District staff contrasted their COE’s approach to that 


of 


an external nonprofit provider with whom they had been working over the previous year. 
This external provider had consistently challenged district staff to go deeper into their data 


and consider alternative explanations or strategies. 


In a third county, district leaders indicated that while the data review with COE staff 
had been helpful, their process had failed to move beyond the initial root cause analysis 


to explore specific levers and strategies to address the identified problems. ‘It felt like 
we started a process and then it just stalled,” said one administrator. Added another, * 
getting to the change ideas and drivers was a little frustrating for me as a participant.. 


Not 


[The county] has a very light touch. | think in one meeting we even acknowledged that 
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there was no plan for action." 

One explanation for the limited depth — mentioned by COE and district respondents 
alike — was the short timeframe for DA, from the release of the Dashboard results in 
December to the COE’s report back to the district by May. We delve into this timing 
problem more deeply in our discussion of the Dashboard. Another factor, also discussed 
below, was the varying capacity of COE staff in districts’ identified areas of need. Finally, 
several respondents pointed simply to the newness of the system and of the COEs’ 
emerging role within it. As one district respondent put it, “This has shifted to an openness 
to try to work with each other. But | think we're all getting our sea legs a little bit in terms 
of ‘what does this mean’?” Several COE respondents voiced similar patience even when 
noting implementation shortcomings and glitches in the state system. Said one county 
superintendent, “I’m not sure if the current [accountability system] is there yet, but | think it’s 
definitely a step in the right direction. It’s a huge system that | think just needs patience 
and support.” 


3. COEs Varied Substantially in Their Interpretations of the COE Role Within the 
System of Support 


Despite the surface similarity of COE actions to implement DA, deeper discussions 
with the leaders in Our case study counties revealed considerable variation in their 
conceptualizations of the goal of differentiated assistance and their role within it. These 
conceptualizations were strongly (and often explicitly) related to the COE leaders’ and 
staff members’ assessment of the overall county context (e.g., size, urbanicity, social and 
economic conditions, and reform history), the level of capacity within the COE relative 
to the needs of their districts, and COE leaders’ beliefs about the change process. We 
highlight this variation because COE perspectives on the goals of DA and their role within 
it influenced the ways that counties organized and staffed the work with their districts and 
could, if studied further, provide lessons about the kinds of support that are more effective 
or less effective in different contexts. At the same time, such differences — while perhaps 
responsive to local context — could also have implications for equity among districts 
across the state in the nature and level of support they receive, especially if approaches 
in some counties prove more effective and districts in other counties simply do not have 
access to those supports. For this reason, such variations in county approach and impact 
will be important to monitor over time. Three examples help to illustrate the range of COE 
approaches to their role. 


County A: The COE as a one-stop shop for DA. In this approach to differentiated 
assistance, the role of the COE is both to support the develooment of continuous 
improvement processes (e.g., root cause analysis, Plan-Do-Study-Act cycles) in eligible 
districts and to provide professional development and other direct services in particular 
content areas (such as middle-grades mathematics, early literacy, or special education) 


edpolicyinca.org & 


VA PACE 


identified through their Dashboard results. One of our sampled counties was particularly 
illustrative of this approach. In this county, each COE team working with one or more DA 
districts is led by an improvement science coach who is deeply steeped in the principles 
and processes of continuous improvement, having trained under the tutelage of Carnegie 
Foundation staff over the course of two years (training that was funded through outside 
grants prior to implementation of the System of Support). 


Beyond the continuous improvement process, this COE also serves as the provider 
of professional development and other support in this isolated rural county. Given the 
small size of the districts in the county and their general isolation, there is little incentive 
for outside providers to invest the time and resources that would be necessary to address 
the significant educational needs deriving from the deep levels of poverty in the region. To 
fulfill this substantive assistance role, county leadership has both recruited new staff and 
encouraged the development of existing staff in most of the topical areas relevant to their 
districts. One reported advantage of this dual role is that the COE is able to take a longer 
view of differentiated assistance than we saw in most other counties. Work in the first year 
of DA is thus designed to be methodical and deep to ensure that district staff understand 
the problem(s) they seek to address. The COE then builds on the foundation laid in year 1 
to help the districts implement and refine the change strategies they have developed. We 
Should note, however, that while this COE took the long view and saw DA as an ongoing 
process, the volatility of the Dashboard with respect to which of its districts were identified 
for DA, and for which student groups, made that approach more difficult to implement, 
especially for small districts. (See box on small districts and the discussion of Dashboard 
volatility below.) 


County B: The COE as a thought partner for continuous improvement and 
culture change. A second approach to differentiated assistance focuses on the processes 
of root cause analysis and continuous improvement, leaving the actual solutions to 
identified problems of practice primarily up to the districts. One COE in our sample had 
determinedly adopted this approach: “[Our role is] asking ‘critical questions’ so the district 
can develop a strategy to address the issues.” Leadership in this COE believes that DA 
is not about bringing expertise on specific subject matter or populations (e.g., special 
education or homeless youth); there are many other resources to which the districts in 
this primarily urban and suburban county can turn for such substantive help. Rather, these 
county leaders believe that DA needs to be organized around facilitating district staff 
to see the big picture, engage in self-reflection, develop an understanding of their own 
data, and create strategies to solve their own problems. In this way, districts will “break 
their compliance mindset” and build the capacity to tackle complex issues that include 
but extend well beyond those for which they are identified for DA. As the superintendent 
remarked, “We can't do DA for districts; we can only provide the process.” This COE’s 
emphasis on expertise in systems analysis and change management Is reflected in their 
recruitment and training of staff providing DA. 
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County C: The COE as both service 
provider and broker. A third approach to 
differentiated assistance arose in one of our 
sampled counties that had a very large geographic 
dispersion of high-needs districts, limited capacity 
in the COE to address them all, and access to at 
east some external groups willing to work with 
districts in the larger metropolitan areas. In this 
situation, the COE adopted an explicit brokering 
approach. COE staff worked with districts on the 
initial root cause analysis and also provided direct 
services in the areas in which they have expertise 
in-house. They then helped districts connect 
with other sources of assistance as needed and 
available. As one COE administer explained, “We 
will outline and define what we know we do well 
and then [are] willing to broker services in the 
areas that we Know we would love to go deep in 
but we Just don't have the capacity and personnel 
on staff.” 


This explicit brokering role is evident in the 
approach that County C has taken to allocating 
the additional support dollars it receives from the 
Budget Act of 2018.18 Because the COE has been 
able to build up a substantial dedicated LCFF fund 
that can cover the basic cost of the root cause 
analysis phase of DA, all these additional monies 
(approximately $4 million) can go to support the 
actions that the districts decide to take up based 
on that analysis.? As the assistant superintendent 


Survey Results: 
Brokering Support is 
Gaining Popularity 


The notion that a viable approach 
for COEs is to serve in part as a 
broker, connecting districts with 
outside resources (County C), 

is one that seems to be picking 
up steam. In January 2019, 48 
percent of COE superintendents 
surveyed for this study agreed 

or strongly agreed with the 
statement: “COEs should help 
districts identify underlying 
causes of performance problems 
and then connect them to 
sources of high-quality expertise 
and assistance, wherever it 
comes from.” This was up from 
18 percent of respondents 
answering the same question in 
October 2017. Meanwhile, only 
235 percent of other surveyed 
COE administrators — those more 
directly involved in the work 

with districts — agreed with the 
superintendents on this question. 
This may indicate a perceived 
pressure by those closer to the 
ground to try to meet all their 
districts’ needs in-house.” 


18 Recognizing that differentiated assistance is "new" work for COEs, the Budget Act of 2018 included additional allocations 
to COEs to support district improvement efforts. This includes a flat $200,000 for each COE serving districts eligible for 
differentiated assistance, and variable funding based on the number of small, medium, and large districts in the county 


identified for DA. 


1 This COE’s efforts to build a dedicated fund for LCFF implementation had been both strategic and long-range. These 
efforts involved taking advantage of additional monies coming into the county due to state equalization, reallocating 

other COE budgetary resources towards LCFF support, and protecting the fund in the face of competing demands and 
budgetary constraints. Sustained commitment by COE leadership was essential. 

0 Plank, D., Humphrey, D.C. & O'Day, J. The Changing Role of County Offices of Education: Survey Results. Policy Analysis 
for California Education. December 2019. Access at https://edpolicyinca.org/publications/changing-roles-county-offices- 
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explained, “The districts make requests of certain amounts of money to pay for key 
activities or training or technical assistance that they felt would be most useful to them... 
We worked with them to be sure it made sense and that their requests were aligned with 
the LCAP, but it’s their judgment.” The COE then contracts with the outside provider the 
district has selected and ensures that the services are delivered as promised. 


4. COE Capacity Remains a Concern Despite Efforts to Increase COE Effectiveness 


Perhaps the most controversial and yet pervasive question about the design of 
the Statewide System of Support has been its reliance on the COEs as the first line of 
support for districts identified for differentiated assistance. Central to this concern among 
stakeholders has been the observation that COEs differ significantly with respect to their 
ability to meet the many and varied needs of their districts. County leaders themselves 
are aware that the shift in their role places new demands on their staff and agency 
organization. In essence, COEs are being asked to reinvent themselves, which requires 
considerable attention to internal capacity building. For this reason, all case study COESs 
had actively taken steps to build their capacity through training and hiring staff and 
through restructuring their organization. 


COE efforts to increase staff expertise through training and recruitment. COE 
staff have sought and received training in continuous improvement processes or systems 
analysis. In one case, the leads for each of the DA teams had participated in 24 days 
of improvement science training over a two-year period with the Carnegie Foundation 
for the Advancement of Teaching. This extended period allowed staff to deepen their 
understanding of improvement methods through an iterative process of study and 
application. Lead staff in other counties also reported receiving some — though generally 
far less extensive — training on root cause analysis and other aspects of continuous 
improvement, provided either by the Carnegie Foundation (through CCSESA) or WestEd. 
Most COEs had also recruited new staff to address substantive gaps in their expertise. 
Two case study counties were recruiting heavily at the time of our visit; the others had 
previously made significant staffing changes. One county (County C above) had done no 
recent recruitment but stretched their ability to support their districts by connecting them 
with expertise and sources of assistance outside the COE. 


Implementation of cross-functional teams to improve coordination and 
effectiveness. |n addition to staffing changes, all the case study COEs had internally 
restructured to increase effectiveness and improve their relationships with districts. 

Each was using cross-functional teams as an integral part of their DA work, although 
composition of the teams differed across the counties. In two COESs, the structure for DA 
was simply an extension of the one they had developed for the LCAP, with particular 
COE staff being assigned to work with a set of districts on an ongoing basis. Staff in these 


The Early Implementation of California's System of Support 


Policy Analysis for California Education 


COEs explained that this approach allowed them to get to know the districts’ work more 
deeply, establish relationships with district staff, and link the differentiated assistance 
process more directly with LCAP development. 


Regardless of team genesis, all COEs reported an intentional fluidity to staff 
participation in the DA teams in order to respond to particular needs of the districts, such 
as data analysis, special education, school climate, or curriculum. One COE administrator 
noted: 


We created [the improvement teams] but the goal was to have representation 
from different groups and it kind of evolved as we started... When [the district] 
chose a focus, we would change our groups sometimes to make sure we 
had county-level expertise in that area of focus. 


Sometimes, however, this fluidity may have been too pronounced. In one of the 
COEs, the size of the overall DA team and the complexity of assigning staff to districts 
(coupled with COE staff turnover) left districts unsure about who their main county 
contact for DA even was. 


Survey Results: COE Self-Assessments 


Given the focus on continuous improvement training, it is not surprising that survey responses 
from both county superintendents and assistant superintendents reported this to be an area 

of relatively high capacity in their COEs. Asked to rate on a scale from 1 (lowest) to 10 (highest), 
how prepared their county was to provide assistance in district-level root cause analysis, 

80 percent of superintendents rated their COE at level 7 or above. This was substantially higher 
than their self-ratings in any other area. Using the same threshold of a 7 rating or higher, only 
57-61 percent of superintendents gave their counties high marks for expertise in other key 
areas, including aligning instructional practice with standards, reducing chronic absenteeism 
and suspensions, improving graduation rates, and data systems and data analysis. Moreover, 

at or near the bottom of the self-ratings were two areas particularly relevant to the System 

of Support: evaluating the efficacy of interventions (an important aspect of continuous 
improvement) and building district structures to support school improvement (an underlying 
justification for using districts as the unit of change in the accountability system). These came in 
at 40 and 49 percent respectively. 


= 


+t Plank, D., Humphrey, D.C. & O'Day, J. The Changing Role of County Offices of Education: Survey Results. Policy Analysis 
for California Education. December 2019. Access at https://edpolicyinca.org/publications/changing-roles-county-offices- 
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District perceptions of COE capacity. Our data suggest that district perceptions of 
the usefulness of DA was strongly conditioned on their perceptions of the capacity of their 
COEs to understand and effectively respond to the needs and challenges of their local 
district. In two of our sampled counties, perceptions of both capacity and usefulness were 
almost uniformly positive. In the two counties in which districts were least sanguine about 
differentiated assistance, however, concerns about COE capacity were central: 


| have to be honest, | do not call them. Individuals at the county office are 
very nice, but some of the information | have received from them has been 
inaccurate. They are accessible, they want to talk. But a lot of their answers 
are, ‘Let me get back to you.’ 


| would say they don't have the capacity... They have some pretty good 
people... but it’s going to take an outside agency to help in that area. 


Perceptions of county capacity varied within as well as across counties. More 
specifically, we observed that even when faced with similar performance challenges, 
districts within the same county often had quite different perceptions of their county's 
ability to assist them. Two main factors appeared to influence this variation in district 
perceptions: the district's size and/or level of capacity and its prior experience with 
continuous improvement.” 


Influence of districts’ size and capacity. We observed that in these five counties, 
the larger and/or higher-capacity districts often had more internal and external resources 
to draw on than did their COEs. Some of these districts appeared to find limited added 
value from the short sessions On root cause analysis with county staff. One district 
superintendent expressed this perspective clearly: “| don't think the county has any 
particular skill set we don't have internally.” In contrast, some of the smaller, more isolated, 
and lower-capacity districts relied on the support they received from their county offices, 
and at least one COE in our sample tended to focus their attention and resources on 
these smaller districts. This suggests that the System of Support as currently designed 
may be more relevant for the 40 percent of small districts across the state for whom the 
county provides the main source of intellectual and professional challenge and support.*% 
(See Small School Districts box.) 


*2 Note that we had no independent means of assessing the actual capacity of COE staff, so we focus here on district staff 
perceptions. 

*3 "Currently, about 40 percent of public school districts in California are “small” (serving fewer than 1,000 students), and 
about 10 percent of all districts are "very small" (serving fewer than 100 students).” https://lao.ca.gov/reports/2011/edu/ 
district_consolidation/district_consolidation_050211.aspx 
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Influence of districts’ prior experience with continuous improvement. |n addition 
to overall district capacity, the degree to which districts have had prior exposure to and 
involvement in continuous improvement processes seems to have played a significant 
role in their receptiveness to county assistance. At least six of our sampled districts had 
been previously (and in some cases, were currently) involved in continuous improvement 
collaborations with other districts and external partners prior to the implementation of 
differentiated assistance. These districts had already analyzed many of their data and 
may even have gone through several Plan-Do-Study-Act (PDSA) cycles of inquiry and 
improvement around an identified problem of practice. So to the extent that their COE 
presented a process that started again at square one, it may have seemed like a waste 
of time that would not move the district beyond what they had already learned and 
accomplished. Some called the process “duplicative” or “repetitive” and thus of less value. 
The words of one district respondent were echoed by many others: “We feel like we are 
getting more from working with other districts with like needs than we have from the 
county.” This pattern has several implications: 1) COEs should tailor their approach to DA 
to not only the Dashboard data but also the varied experiences of the districts, including 
their prior knowledge of continuous improvement processes; 2) the most beneficial 
assistance for some districts may be to continue the work they are already doing in 
outside partnerships, and this work should be encouraged and supported financially. 


5. Both COE and District Respondents Raised Concerns about the Under-resourcing 
of the Support System Relative to the Needs in their Districts 


Included in the design of the new system were monies to support COEs In 
helping their identified districts. The amount of these funds varied by the size of the DA- 
eligible districts, ranging from $100,000 to $300,000 per district, and were available for 
use at the discretion of the COE. With the exception of County C above, all other COEs 
in our sample used their DA funding to pay for the root cause analysis process and to 
offer scholarships for — or a specified number of days of — county-led professional 
development. Two COEs also provided a small stioend ($35,000 to $75,000) directly to 
districts to use for improvement actions of their choice (aligned to the LCAPs). 


Overall, however, there was a consistent recognition in the districts and the counties 
that there were just too few additional funds to support the needed improvement. Said 
one district respondent, “With the API... if you were in improvement there was money. 
Here with the Dashboard, it is really about improvement, and | am excited about that. The 
issue Is that there is no money.” Our surveys of county administrators suggest that COE 
Staff share a similar financial concern about the new system. When asked, 77 percent of 
county superintendents and 61 percent of assistant superintendents disagreed or strongly 
disagreed with the statement “Our COE has adequate funding to implement the System 
of Support in the districts in my county.” Indeed, the funding to fuel improvement efforts 
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appears even smaller when considered in light of the emerging economic climate facing 
many districts today, as the superintendent of one of our case study counties pointed out: 


The first few years of local control [were spent] restoring the programs they 
had had to cut... The next two years | started to see some really creative 
ideas and things moving forward that could have really made some impact. 
And then all of that was put on a halt with the pension increases.~ So, 
districts are really losing now. They don't have the resources to continue 
their efforts because these new expenses are draining their budgets... 

You can build a great model, but if you don't have the means to execute, 
you dont get anywhere with that model. 


In addition, when defining resources to include supports beyond money, both 
COE and district officials reported that they were unclear about the roles of the CDE and 
the CCEE in the System of Support. As a result, our interviewees did not view these state 
agencies as important resources on which to draw for improvement. 


District and county practitioners also raised concerns about other issues that 
they perceived as beyond their control but that greatly influence the potential success 
of the System of Support. These include the timing of the various components of the 
system (particularly the LCAP and the Dashboard), aspects of the Dashboard design, 
and the overall insufficient state infrastructure to address problems in special education. 
We discuss these concerns below, focusing on the design and implementation of the 
Dashboard, which triggers differentiated assistance. 


The Dashboard and the System of Support 


The implementation of the Statewide System of Support — particularly with respect 
to differentiated assistance — occurs largely through the actions and interactions of COEs 
and their districts, albeit with help from an expanded set of other state agencies. We have 
noted previously how local context and capacity influence the decisions that county 
and district actors make in the definition and exercise of their roles. Equally important, 
however, are the ways in which the broader policy context facilitates or constrains their 
support and improvement efforts. As one state policymaker told us, the System of Support 
was built to support and be integrated with all the elements of the LCFF. “It’s all part of the 
accountability structure.” 


*4 For an examination of the impact of the pension increases on districts in California, see H. Melnicoe et al., The Big 
Squeeze: How Unfunded Pension Costs Threaten Educational Equity. https://www.pivotlearning.org/wp-content/uploads/ 
2019/05/the-big-squeeze-report-april-2019.pdf. 
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A core element of that larger structure is the Dashboard, the intended purpose of 
which is not only to identify districts that are eligible for DA but also to provide evidence 
of progress and help ensure attention to targeted student groups and to state priority 
areas. In this study, therefore, we sought to understand how counties and districts 
are interpreting and using the Dashboard to guide improvement efforts. How was the 
Dashboard helping them focus on key problem areas and populations? How was it 
helping them to understand the systemic patterns underlying those problems and to test 
out and refine solutions based on evidence? 


Unlike the previous system, the Dashboard rejects the assumption that schools and 
districts can be Judged by a single number. As described earlier in this report, the Dashboard 
measures districts and schools’ performance based on academic outcomes, attendance, 
suspensions and expulsions, college and career education, graduation rates, and English 
learners’ progress, along with a variety of local indicators that address the state priorities. 


County Office and District Officials Have Embraced the Dashboard as 
an Improvement 


COE superintendents, staff, and district officials appear to hold positive views 
of the new Dashboard as an improvement over the previous measures of school and 
district performance. Our survey of COE superintendents and COE staff serving on the 
Curriculum and Instruction Steering Committee (CISC of CCSESA) showed strong support 
for the new System of Support, and our interviews with COE superintendents and staff and 
district superintendents and staff also revealed generally positive views of the Dashboard. 


While we heard some concern that the complexity of the Dashboard was a barrier 
to parents’ and other stakeholders’ understanding of district and school performance, 
recent changes to the Dashboard's presentation were generally seen as an improvement. 
A typical assessment from a district superintendents was: “I actually think the Dashboard 
has a lot of promise.” As one principal said: 


For the first time in my career, finally the data feels like a true representation 
of our school. The Dashboard allows you to see the growth and positive 
areas, in addition to bad areas. For example, we're doing a great job in CCR 
[College and Career Readiness], but you never would have seen that before 
the Dashboard. 


Perhaps the most notable feature of the Dashboard that garnered support from 
district and COE Officials was its ability to identify the performance of student subgroups 
across multiple indicators. In particular, many of our interviews reported that the 
Dashboard has shone a bright light on the poor performance of students with disabilities. 
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As one COE Official told us, “The boundaries have come down... students with disabilities 
are seen as everybody's students... because of the Dashboard and DA.” (See Students with 
Disabilities box). 


Voters’ and parents’ responses on a recent poll conducted by PACE suggest that 
more work Is needed to increase Dashboard awareness among the public, although 
those somewhat familiar with the newer version of the Dashboard have generally positive 
assessments of it. In 2019, 46 percent of voters and 63 percent of parents reported general 
awareness of the Dashboard, but only 9 percent of voters and 19 percent of parents 
reported that they knew a lot about it.2° At the same time, among those respondents who 
were shown the old version and the new version of the Dashboard, 65 percent of voters 
and 81 percent of parents expressed a positive impression of the new version. 


Both COE and District Officials Pointed to Problems with the Dashboard that 
Constrain or Undermine the Effectiveness of the System of Support 


Educators generally appreciated the Dashboard as a more accurate measure of 
performance; however, they expressed numerous concerns about some features of 
the Dashboard. Our interviewees argued that the problems with the Dashboard actually 
impeded the successful implementation of the System of Support. These problems result 
in disincentives to mount a sustained improvement strategy; the lack of timely, valid, 
and comprehensible measurement to guide improvement efforts; concerns about the 
equitable distribution of support; and a truncated timeline for improvement efforts. 


The Dashboard's Multiple Indicators and Volatility Undermine the Focus Needed to 
Sustain Improvement 


Too many priorities. Many of our interviewees argued that the state established 
too many priorities for districts to address simultaneously. As a result, the Dashboard can 
identify a district for DA due to the performance of one among many subgroups across 
any two of the six priorities currently measured. A common concern articulated by one 
COE superintendent was: “The priorities are indigestible because there are just too many 
of them.” This observation echoes the findings from the recent report by Fullan, Rincon- 
Gallardo, and Gallagher (2019). As they reported: 


In the coming years, the number of districts identified for differentiated 
assistance, and the areas where they will require support might increase 


* Polikoff, M. S., Hough, H., Marsh, J. A., Plank, D. N. (2019, February). Californians and Public Education: Views from the 
2019 PACE/USC Rossier Poll. Policy Analysis for California Education. Access at: https://edpolicyinca.org/publications/ 
pace-and-usc-rossier-polls-2019 
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Students with Disabilities and the System of Support 


The 2018 Dashboard identified 374 districts eligible for differentiated assistance; 2435 of those 
were identified because of the poor performance of their students with disabilities (SWDs). 
Although the numbers were lower in 2017, the pattern was the same, prompting our case study 
counties to take steps to increase their internal expertise to help their districts better serve this 
population of students. Meanwhile, the state has established seven lead agencies among the 
Special Education Local Plan Areas (SELPAs) to share expertise and build capacity across all 
SELPAs in both system improvement and various content areas. 


Despite these actions, our study respondents raised many concerns about how the System 

of Support can address the needs of SWDs. We found widespread disappointment among 
state, COE, and district officials about the lack of progress since the 2015 Report of California's 
Statewide Task Force on Special Education. The Report called out issues regarding early 
learning, classroom practices, educator preparation and professional learning, assessment, 
accountability, family and student engagement, and financing. The Task Force made concrete 
recommendations in an effort to create one system to serve all students, but only marginal 
policy changes addressing the needs of SWDs have been enacted. As one policymaker 
explained: "! feel like special ed is a wing of the house that nobody's opened the door to ina 
long time... It’s always felt like the stepchild of the system." 


In addition to the many seemingly intractable problems articulated by the Task Force report, 
COE and district respondents responsible for SWDs also noted increases in the number of 
students needing social and emotional support and students on the autism spectrum. Some 
warned of the unintended consequences of the recent increase in compliance reporting and 
the lengthening of Individual Education Plans (IEPs). As one official argued, “The paperwork 
documentation in the special ed world from IEPs and these audits... It's going to crumble the 
whole system.” Others focused on the shortage of special education teachers (especially in the 
rural districts), although some argued that the problem was that credentialed special education 
teachers often move to regular teaching assignments due to the burdens of the job. 


Our case study COEs were working to address these issues by helping their districts examine 
their systems and develop more efficient and effective approaches. But COE officials readily 
admitted that their efforts cannot begin to address the fundamental structural problems districts 
face in addressing the multiple challenges of educating their students with disabilities. To the 
state's credit, AB 75 requires a number of actions to improve planning, coordination, and the 
delivery of services for SWDs. The Dashboard results have shone a spotlight on California’s 
failure to adequately address the needs of our students with disabilities, but it will take system 
transformation at all levels to significantly improve both opportunities and outcomes for this 
large and diverse population of students. 
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at a faster pace than the system can respond. This creates the risk of 
scatter. Eight state priorities are too many. Adopting multiple measures 
and reporting results by student group were important steps, but the 
accumulation of new constituents for each state priority can create 
incoherence and confusion.”¢ 


Other COE and district superintendents reinforced this finding. As one COE 
superintendent stated, “...the districts’ resources are finite and so are ours. So how do you 
make the choices about what to emphasize? ...it is really too many things to focus on at 
one time and do it thoughtfully.” 


One leading advocate and observer we interviewed concurred with this 
assessment. As he argued, “We prioritize too many things. ...| would say we need to create 
three indexes. ...Just create one that’s academic, one that is a social-emotional learning 
(SEL). ...and one that is equity.” 


The eight priorities are important to the state and none of our informants suggested 
that they were not. The informants did argue, however, that COEs and districts could use 
more flexibility to focus on a fewer number at a time. 


Multiple indicators, not multiple measures. California policymakers are rightfully 
proud of moving beyond the single numerical measure of school and district performance 
(the Academic Performance Indicator (API)). We found general support for the new 
measurement system among COE and district leaders, but also heard concerns about the 
misuse of the term “multiple measures” when describing the Dashboard. As one union 
Official noted, teachers’ perspective on what constitutes multiple measures is very different 
from the state's use of the term: 


..In education, what that [multiple measures] means is you have multiple 
indicators for a single thing. | have multiple ways to identify how my 
students are doing in reading comprehension. But what the state did 
was they picked multiple areas [for which] to have single indicators. ... 

it causes people to overly simplify what the problem is and, based on the 
oversimplification of the problem, overly simplify the solution. 


Several of our interviewees agreed that the Dashboard can appear to be six APIs 
rather than the kind of multiple measures teachers typically employ in their teaching. While 
the state’s move to multiple indicators has been welcomed as a clear improvement over 


6 Fullan, M., Rincon-Gallardo, S., Gallagher, M. J., (June 2019). Learning is the Work. Motion Leadership, p. i. 


The Early Implementation of California's System of Support 


Policy Analysis for California Education 


the API, the Dashboard does not overcome the danger of oversimplification of problems 
and solutions. Moreover, the multiple indicators can be seen as disparate aspects of 
school performance, leading to the pursuit of strategies to fix one particular indicator 
one year and a different one the next. As we discuss next, the volatility of the Dashboard 
compounds this problem. 


Volatility. Tne 2017 Dashboard release identified 226 school districts for 
differentiated assistance. With the addition of chronic absenteeism and students’ readiness 
for college and careers indicators, 374 school districts were identified for DA in the 2018 
release. Among the district officials in our sample, many expressed concern over the 
apparent volatility and the large increase in the number of subgroups that fell into the 
lowest rating from 2017 to 2018. One district superintendent described the problem as 
“whack-a-mole.” As the superintendent explained, one year the district was identified for 
assistance based on the performance of two groups, and the next year the district was 
identified for two different groups. Several COE superintendents agreed that the volatility 
of the Dashboard was problematic, noting that the annual fluctuations are often a result of 
the changing numbers of students in various subgroups from year to year. Having different 
subgroups emerge as deficient from one year to the next makes it hard to value the data 
and mount a sustained focus. 


Officials in small school districts were particularly concerned about volatility 
issues because of the low and fluctuating numbers of students in any given subgroup. 
In response to these concerns, the state has reduced the number of change levels for 
graduation, suspension, and chronic absenteeism indictors for small districts. Small 
districts’ performance on these indicators is now rated as increased, maintained, or 
declined (thereby dropping the increased significantly and declined significantly ratings). 
With fewer ratings of change, the Dashboard’s volatility for small districts should be 
reduced. Despite these changes, our interviews with officials from small rural districts 
reported that the Dashboard fails to accurately identify the major problems they face. (See 
Small School Districts box). 


The Dashboard's Technical Problems Undermine the Validity and Comprehensibility 
of Measures to Guide Improvement Efforts 


More technically oriented officials in the districts and some researchers expressed 
concerns about the basic structure of the Dashboard. While agreeing with the use of both 
change and status measures to determine district and school performance, some officials 
stressed that the cut scores on the status measures are very low. As one administrator 
reported, “I think that the state did make an error in the Dashboard. The biggest error [is 
that] they set their cut points too low in some cases.” 
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California's Small School Districts, the System of Support, 
and the Dashboard 


Of all of the districts in Our sample, the majority of officials in small districts were most appreciative 
of the assistance they received from their COE. While there were significant differences in the 

size of the small districts in our sample (ranging from 100 to 1,000 students), their administrators 
viewed their COE as the primary source for quality professional development and technical advice. 
As one small district administrator reported, “[Our COE] has hired experienced knowledgeable 
personnel who can easily come in and work with a staff... and do it in a way that’s not ‘shame on 
you’ but instead ‘let's roll up our sleeves...” This view is consistent with previous reports of small 
district staff as they worked with their COE to implement the LCFF and develop their LCAP.”” 


These positive views come in the context of our case study districts with mostly poor and white 
student populations and pockets of Native Americans. Chronic absenteeism and suspensions 
are common. District leaders reported that they had increasing proportions of students facing 
significant trauma related to poverty, substance abuse, and homelessness. Citing students’ need 
for significant social and emotional support, one administrator noted, “Kids first need to get 

to school and get equipped with supports so they can learn in the first place.” Compounding 
these challenges, some families and educators in the smallest districts were highly suspicious 
of government institutions. In addition, some district administrators in the small, “off the grid” 
districts did not fully buy into the System of Support or believe that the state cared enough 
about their challenges to make submitting their data worth the effort. 


Both COE and district officials reported that limitations of the Dashboard made it “irrelevant” or 
“inaccurate” for many small districts. Administrators in the smallest districts argued that there are 
great needs that aren't being accounted for by the Dashboard and that some districts should 

be eligible for differentiated assistance but are not. In addition, some small districts have too 
few students in subgroups to be reported on any of the six indicators and some had problems 
submitting their data. As one administrator reported: "...staff are strapped, they are slow to learn 
CALPADS (California Longitudinal Pupil Achievement Data System)... There are multiple Student 
Information Systems and we have not done the best job scrambling with the CALPADS system. 
[And] there is lots of turnover in the smalls.” In one county, approximately one in five districts did 
not enter their Dashboard data, and 40 percent of districts entered their data but their student 
groups were too small to show up on the Dashboard. 


The state's reasonable concerns about student privacy and the impact of a few very low- or 
high-performing students on performance ratings mean that indicator results are not reported 
for student groups numbering fewer than 11, and groups numbering 11-29 students are not 
assigned a color-coded performance level. While the state has taken steps to reduce the 
volatility of the Dashboard, district officials found the Dashboard to be of limited utility, leaving 
them on their own to identify student groups that need attention. 
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Noting that low cut scores likely stem from political issues, another administrator 
responsible for analyzing district data surmised that: 


..the [California] State Board [of Education] didn’t want too many schools 
in red and orange so when they set their cut points they didn't set them 
on what's really our target. ...!'m fearful of the day when they adjust the cut 
points and what that’s going to do to the system. Because it’s almost like 
we were good enough now but then we're not going to be good enough. 
I'm not sure what message that sends to the schools about what we really 
want. 


For most educators and the public, recent changes to the Dashboard have made it 
more comprehensible, but the technical intricacies are beyond the understanding of most. 
For example, it is not easy to interpret what having a Distance-from-Standard (DFS) score 
of 167 negative points actually means. 


Some researchers have expressed additional technical concerns. Polikoff found at 
least four significant limitations of the Dashboard in its current form. First, the Dashboard 
does not allow users to compare schools between districts. He argued that users should 
be able to select as many schools as they would like from across districts and charter 
schools for comparison purposes. Second, the Dashboard does not allow comparisons 
across multiple indicators and among schools with similar demographics. Third, he found 
that the Dashboard allows users to compare schools only on the overall performance 
of each indicator, but not on student group performance. Finally, Polikoff argued that 
the state’s approach to student growth is simplistic and ignores cohort effects, and thus 
should be replaced with a more appropriate growth measure (Such as a two-step value- 
added model). Notably, only California and Kansas have not adopted a growth model.”8 


Yellow is not yellow. Some district officials also reported concerns about the 
meaning and lack of clarity of the colors on the Dashboard. Because the colors represent 
both status and change measures, a high-performance district or school could be in the 
middle range (yellow) because it declined on a given measure relative to the previous 
year, while a low-performance district could earn the same yellow designation by showing 
positive change despite low status. As one district data official pointed out: 


*” Koppich, J. E., and Humphrey, D. C. (2018, September). The Local Control Funding Formula: What Have We Learned 
After Four Years of Implementation? Policy Analysis for California Education: Getting Down to Facts II. Access at: 
https://gettingdowntofacts.com/publications/local-control-funding-formula-after-four-years-what-do-we-know 

8 Polikoff, M. S., (March 2019). Gauging the Revised California School Dashboard: Evidence form the 2019 PACE/USC 
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Yellow is not yellow is not yellow. ..vou can be yellow because you are kind 
of in the middle of doing nothing. You can be yellow because you're high 
but went backwards. And you can be yellow because you're low but you're 
increasing. And we're seeing a lot of our Title | schools [are not] just red 
and orange anymore. They're getting into that yellow, and why? Because 
their change is growth. And | got some of my high-end schools actually 
that arent [blue] anymore... They're going into green and sometimes... into 
yellow because they are really high but they're seeing that decline. 


As this administrator explained, the upside of this is that some low-performing 
schools are suddenly able to celebrate getting a yellow rating, even though their status 
performance Is still extremely low. State officials acknowledged the yellow-is-not-yellow 
issue, but argued that Dashboard users can dig deeper and see where each district and 
school landed on both status and growth measures. 


The State's Definition of English Learners May Limit the Dashboard's Ability to 
Accurately Identify Districts Needing DA for Current and Long-term English Learners 


Several district officials argued that English learners are being shortchanged by the 
Dashboard and the state’s definition of this subgroup of students. As one superintendent 
argued: 


..the decision was made at the state level to put English Learners and 
Reclassified English Learners [ELO and RFEP] ... together. So, it looks like on 
average... that people are doing well. You miss the needs of the kids who 
are English Learners still and particularly long-term English learners. ... There's 
no dedicated funding, there's no dedicated professional development to 
address those needs of English Learners. 


Researchers concentrating on English learners have elaborated on this concern. 
As they argue: 


The combined ELO + RFEP subgroup resulted in the vast majority of districts 
falling within the Yellow, Green, or Blue bands in the Academic Indicator 

for ELs. These results could potentially fail to address the needs of ELs and 
exclude them from receiving technical assistance and financial support in 
their LCAPs.#? 


*° Lavadenz, M., Armas, E., and Hodge, S. J. (2018). Masking the Focus on English Learners: The Consequences of 
California's Accountability System Dashboard. Results on Year 4 Local Control and Accountability Plans (LCAPs). 
Californians Together and Loyola Marymount University. P. 7. 
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According to the researchers, the aggregated results from the Dashboard could 
mask the results for English Learner Only (ELO) students, including the various EL 
profiles (i.e., long-term English learners, newcomers, etc.). The researchers argued that 
obscuring ELO results could have detrimental effects on districts’ abilities to address 
LCAP goals, set growth targets, focus programs and services, and allocate supplemental 
and concentration funds for this targeted group of students. Of course, knowing 
the performance of reclassified as well as ELO students gives some indication of the 
effectiveness of schools and districts with this population as a whole. But combining the 
two groups has raised equity concerns among some researchers and advocates. 


The Timing of the Release of the Dashboard Presents a Key Implementation Problem 


California collects mountains of data from districts, but not all of those data 
are available in a timely fashion. In order for the state to ensure the accuracy of all of 
the measures before they go public, the release of the Dashboard currently does not 
occur until December, which is already well into the school year. By the time the COEs 
contact the districts identified for DA, schedule meetings with the districts, and develop 
a plan for assistance, their work with the districts generally does not begin until late 
January or February. Given that COE superintendents are expected to report back to 
the districts regarding their DA by June, DA can feel like it ends right after it begins. CDE 
officials acknowledge the timing problem, but point out that the data on some indicators 
(graduation rates, test scores, etc.) are not available until late August. While there are 
efforts underway to release earlier, realistically the Dashboard is unlikely to be made 
available until the fall. As one COE administrator explained: 


We get the data in December, ...probably letters go out January, do our data 
meetings early February, but that really truncates the timetable for the three 
or four meetings, or five depending on the aistrict... 


Timing of the Dashboard’s release and the ensuing provision of DA also creates a 
mismatch with the timing of the districts’ strategic planning and the LCAP development 
process. Furthermore, district officials reported that by the time the Dashboard data 
are released, the data are a little too old to be helpful. Moreover, districts with more 
sophisticated data systems are able to access more timely data than the state provides, 
rendering the state data less valuable for planning purposes. Tne Dashboard also comes 
out after most districts are already working on their Local Control Accountability Plan 
(LCAP) for the subsequent year.°° As one district superintendent explained: "We need that 


5° We note that district officials continue to argue, as they have since it was first introduced, that the LCAP template, 
despite revisions, is cumbersome and fails to serve its multiple purposes (planning, accountability, and engagement). The 
State Board of Education will release the fourth revision of the LCAP in January 2020, for use in the 2020-21 school year. 
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information sooner. It would be almost like... build a budget without knowing how much 
money you have.” 


Although districts with sophisticated local data systems generally view the 
Dashboard as an improvement over previous systems, they report that it is still inadequate 
to meet their needs. Instead, they pay more attention to more timely information from 
their own local data systems. As one district official reported: 


..the timing Is so late in terms of the improvement cycle. The good news 
is we now get the underlying data earlier than we ever used to. ...By the 
time the Dashboard actually comes out, are we using that data? Yes. Is the 
Dashboard necessarily the driving force? No, because it’s too late in the 
improvement cycle. 


While the timing problems associated with the Dashboard appear to limit the 
effectiveness of DA, our case study COEs were working to expand the support they 
provide, in part by better integrating DA with their other ongoing work with all of the 
districts in their counties. 


Not Yet a System 


A system is usually thought of as a collection of parts or organizations that share 
a common purpose, act rationally and interdependently to achieve the purpose, and 
form a coherent whole. California's System of Support and the state’s overall approach to 
accountability have yet to adhere to all aspects of the definition of a system. As one COE 
leader argued: 


California has been really forward thinking, but... when you look at this thing 
as a system, it felt like it came out in parts... | think sometimes it was just 
“well | have an LCAP and apparently now | have a Dashboard too" instead of 
saying there's a system here that’s leading me towards something, and how 
do we work within that system? 


One reason the state’s approach currently falls short of cohering as a system is that 
while the various governmental and non-governmental organizations playing important 
roles in it appear to share a Common purpose, the specific responsibilities of the CCEE and 
the CDE are not well understood at the local level. And while CDE and COE leadership have 
clearly signaled their commitment to improvement through support rather than compliance, 
both CDE and the COEs still have significant compliance monitoring responsibilities of 
federal and state programs that can send conflicting messages to the districts. 
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Another source of incoherence has to do with the timing of the Dashboard 
release, which, as noted above, is out of sync with timing of districts’ planning and the 
development of their LCAPs. While districts appear to incorporate Dashboard data in 
their LCAPs, most districts’ planning for the coming school year typically is part of LCAP 
development and continues into the summer. Moreover, districts with more sophisticated 
data systems are able to access more timely data than the state provides, rendering the 
state data less valuable for planning purposes. 


A third problem lies in the disjuncture in the sources of accountability between 
the LCAP process and the System of Support. California's accountability system includes 
two somewhat different approaches. In the System of Support, accountability is driven 
by the state identifying district deficiencies via the Dashboard; in the LCAP process, it is 
parents, Community members, and other stakeholders who hold the district accountable 
for meeting goals and engaging In activities set by the local community. Most districts’ 
LCAPs appear to include Dashboard data, but we found few districts or COEs that alerted 
parents and community members of Dashboard results when they were released. Unless 
parents and community members access the CDE website, they are unlikely to be aware 
of the Dashboard results of their district or school. This is understandable, as there are 
no incentives for districts to advertise poor performance and no incentives for COEs to 
undermine positive relationships with their districts by displaying Dashboard results. That 
said, a critical assumption of the LCFF is that stakeholder engagement is a key part of the 
accountability system, and It is appears that the System of Support largely avoids such 
engagement. 


A system also requires adequate resources and the capacity to make strategic 
use of those resources. COEs have additional monies from the state to assist their 
districts as part of the System of Support, but these resources are modest compared 
to the expectation that COEs will help lift hundreds of districts out of identification for 
differentiated assistance. Indeed, the state allocated about $66 million to COEs and about 
$21 million to the Special Education Local Plan Area leads, the GEO and other leads for 
the System of Support for the 2019-20 school year out of a total K-12 education budget of 
$84.5 billion. Even with the additional funds, the COEs do not know how much they will 
receive until the Dashboard is released in December making it difficult to ensure they have 
adequate staffing in place. 


Finally, the System of Support does not systematically mobilize or coordinate 
governmental and non-governmental resources to support district improvement. 
In particular, differentiated assistance Is largely dependent on COEs, which, as we 
have noted, vary in their capacity to respond to district needs. Meanwhile, nonprofit 
organizations, internal expertise from exemplary districts, and institutions of higher 
education (IHEs) must rely on the COEs’ willingness to share their new resources or 
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include them in assistance plans. We found some examples of COEs connecting districts 
with sources of assistance, but that approach was the exception. In general, districts are 
left on their own to access help beyond that provided by COEs. This is the case despite 
policymakers’ stated intention in California's ESSA Consolidated State Plan: 


Critical roles will also be played by multiple stakeholders in the full System 
of Support including other state entities (i.e., the California Commission on 
Teacher Credentialing and California Subject Matter Project), labor, state 
associations, researchers, non-profit organizations, institutions of higher 
education, philanthropy, and coalitions. Systematic collaboration and 
coordination among all of these entities will facilitate coherent technical 
assistance and support at the local level and ensure alignment of efforts to 
continuously improve student outcomes.* 


All of this raises questions about the adequacy of resources, the best way to 
distribute support dollars, and how to marshal all of the expertise in the state in service of 
improvement for all districts and schools. 


Recommendations 


California's new System of Support is in its early implementation stage, and 
modifications are likely to be made as the system matures. This report was designed to 
identify both early successes and challenges. With the intention of maintaining those 
successes and addressing the challenges, we present five recommendations drawn from 
Our research. 


1. The System of Support should include the full complement of expertise in the state, 
including nonprofits, within-district teams, and universities, along with COEs to 
assist districts. 


The central actors in the System of Support are COEs, but there is a huge variation 
in the size, expertise, and responsibilities of COEs. For example, the Los Angeles COE 
serves 80 school districts with 1.5 million K-12 students and about 1,600 staff members, 
while the Alpine COE serves 80 students with five staff members. While the System of 
Support relies on the COEs to assist districts, and to a lesser extent the CCEE and the 
CDE, other sources of expertise are only involved in the support system when the COEs 
are willing to divert their funds (some do; most don't). To create a more robust System 


+t State Board of Education and the California Department of Education (2017). California ESSA Consolidated State Plan. 
https://www.cde.ca.gov/re/es/documents/essastateplan2018.pdf 
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of Support, we urge that the state more effectively and consistently draw on the entire 
Capacity within the state to foster improvement in all districts and schools. There is no 
quick fix to building meaningful support for district improvement across the state, but 
incentivizing COEs to draw on regional expertise from a variety of sources and investing 
resources In exemplary districts to share their expertise in service of other districts could 
lead to a much more effective statewide system. 


2. The System of Support needs to be a sustained, multi-year, and tiered intervention. 


As our research suggests, the current configuration of the System of Support 
suffers from a truncated period of intervention, resulting in a fairly light touch. Some COEs 
have taken steps to provide year-round support as part of the system, but are limited by 
the timing of the Dashboard’s release, inadequate resources, and the brief window for 
their work with districts identified for differentiated assistance. In addition, the System of 
Support provides limited resources to assist districts not identified for DA. Moreover, while 
the System of Support alludes to a more intensive assistance for districts failing to make 
progress after being assigned to DA in three out of four years (Tier 3), the state’s approach 
to Tier 3 support is not clearly communicated. As the System of Support is modified, there 
is a clear need to move beyond a light touch. 


3. The state should make data on each Dashboard indicator available as soon 
as possible to better assist districts’ and schools’ planning and improvement 
processes. 


As district and COE officials emphasized, the timing of the Dashboard release is 
problematic for planning and improvement efforts. While state administrators are working 
to speed up the release, some data elements will never be available before the school 
year begins. Rather than wait until data on all of the Dashboard indicators are ready, more 
timely release of individual indicators as soon as the relevant data are available would help. 
This will require expanding the capacity of CDE’s Analysis, Measurement and Reporting 
Division. Of course, earlier release of the Dashboard data as it is available has implications 
for how districts are identified for DA and suggests that the state should consider 
extending the number of years DA is provided once a district is identified. 


4. Districts and COEs should provide stakeholders opportunities for meaningful 
engagement in developing strategies for improvement that come out of DA. 


A key feature of the LCFF is the requirement to give parents, Community members, 
and other stakeholders meaningful opportunities to engage in decision-making. However, 
the DA process does not include engagement opportunities. Districts generally do 
not make their constituencies privy to the content of the differentiated assistance they 
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receive; there is no mechanism to engage parents and community members in the work 
of the COEs and the districts required to participate in differentiated assistance. Such 
engagement could be incorporated into existing district engagement efforts. 


5. For the System of Support to be successful, the state must allocate adequate and 
sustained resources. 


“More resources needed" is a common finding of researchers examining California's 
education system, and policymakers clearly tire of this repeated mantra. Resources are 
finite and policymakers must make choices. The current configuration of the System of 
Support can be a lifeline for small districts or just a light touch for most districts. Either 
way, the level of investment is too small to establish a robust statewide support system. 
Policymakers may want to consider a reallocation of funds, but without additional 
revenues and investments, a more comprehensive and effective System of Support is 
unlikely to be realized. 
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